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Abstract — In this paper we optimize constellation sets to be 
used for channels affected by phase noise. The main objective is 
to maximize the achievable mutual information of the constel- 
lation under a given power constraint. The mutual information 
and pragmatic mutual information of a given constellation is 
calculated approximately assuming that both the channel and 
phase noise are white. Then a simulated annealing algorithm is 
used to jointly optimize the constellation and the binary labeling. 
The performance of optimized constellations is compared with 
conventional constellations showing considerable gains in all 
system scenarios. 

I. Introduction 

This paper investigates the constellation design for trans- 
mission over AWGN channels affected by phase noise. Our 
main goal is to show that a significant improvement with 
respect to the conventional QAM and PSK constellations can 
be obtained by optimizing the constellation space considering 
both thermal and phase noise effects in the optimization 
process. The shape of the optimized constellation depends 
strongly on the communication system model, the channel 
model, the power constraints and the objective function to 
be optimized. Even for small signal sets with only 8 signals 
several different constellations can be obtained, each being 
optimal for a given set of parameters. We refer the readers to 
Q, El, and (U and the reference within for a detailed 
study of the algorithms and methods used for constellation 
optimization over the AWGN channel. 

Recently ( (5), El) we have proposed a joint signal-labeling 
optimization scheme for designing constellations which max- 
imize the (pragmatic) achievable mutual information under 
the peak power constraint as a function of signal to noise 
ratio (SNR) using a simulated annealing (SA) algorithm for 
transmission over non linear satellite channels. In this paper we 
use the same algorithm focusing on the channels with phase 
noise, imposing an average power constraint. 

In order to use the SA algorithm we first need to calculate 
the mutual information of the constellation for a channel 
affected by white additive Gaussian and phase noise. The 
problem of calculating the channel capacity impaired by 
additive Gaussian and phase noise is open in general and is an 
active research theme. Some asymptotic results for discrete- 
time channels at high SNR has been proposed by Lapidoth 
in (6| for phase noise both with and without memory. In (7) 



Kats and Shamai show that the capacity-achieving distribution 
of the discrete-time non coherent channel under the average 
power constraint is discrete with an infinite number of mass 
points. Also some upper and lower bounds on channel capacity 
are obtained. In particular, the upper bound is shown to be 
tight for asymptotically high SNR. More recently, Barbieri 
and Colavolpe have used a Monte Carlo simulation method to 
estimate the achievable information rate of the AWGN channel 
with phase noise, modelled as a Wiener process, assuming a 
uniformly distributed input [!8j and (9). However these results 
are either too complex computationally or valid only in the 
larger limit of constellation points. In ifTUl the authors calcu- 
late the mutual information for any given constellation with 
uniform distribution over the signal space and assuming the 
memoryless phase noise modelled as Tikhonov distribution. 
In this paper we provide a very close approximation for the 
mutual information which is computationally much faster. 

Optimizing the constellation space under the phase noise 
has achieved a lot of attention in the literature. In ifTTIl a 
64-ary constellation is patented which is robust over the 
phase noise channels. In lfl2l the authors first provide an 
approximate expression for the symbol error rate of a given 
constellation in the presence of phase jitter and then use a 
gradient descent algorithm for finding a constellation which 
minimizes this expression. This approach does not take into 
the account the SNR as an optimization parameter. In the 
same paper the authors also explore the design of circular 
(APSK) constellations for use over phase noise channel. In 
lfl3l the authors optimize the 16-ary APSK constellations in 
the presence of nonlinear phase noise. In the first step the 
authors fix the number of rings and the number of signals over 
each ring. In the second step they find the optimal radii and 
phases of each ring minimizing the symbol error rate following 
the detectors and decoders suggested in lfl4l . 

Our approach in this paper is different from the previous 
works in several ways. First of all we provide an expression 
for the achievable mutual information (AMI) in the presence 
of white phase noise and thermal noise. The AMI is then maxi- 
mized as a function of thermal and phase noise variances under 
a given power constraint. Furthermore, we do not impose 
any particular a-priori structure over the constellation space 
and our method allows for a joint signal-labeling optimization 



when the pragmatic achievable mutual information (PAMI) is 
the target function to be maximized lfl5l . 

The rest of this paper is organized as follows. In section 
ITT1 we describe the notations and formulate the optimization 
problem. An approximate fast method for calculating AMI and 
PAMI in the presence of phase noise is also presented in this 
section. We also provide a short description of the optimization 
algorithm. The optimization results for 8-ary constellations 
are presented in section [Til] where we also provide several 
capacity curves of several optimized constellations. In section 
ITVl we provide end-to-end simulation results of our optimized 
constellations. Finally we conclude the paper in section IV1 and 
discuss some future works. 

II. Statement of the optimization problem 

We consider a complex constellation \ with M = 2 ,n 
elements. The elements of \ are referred as constellation 
points or simply signals. The Euclidean distance between 
two points in the complex plane is denoted by d(.,.), and 
dn(., .) is used to denote the Hamming distance between two 
binary sequences. The signals are associated to the bits at 
the input of the modulator through the one-to-one labeling 
A* : X ~> {0, l} m . In particular, for any given signal x, p % (x) 
is the value of the i th bit of the label associated to it. A labeling 
for x is called a Gray mapping if for any two signal Xi , Xj G \ 
we have d,H = 1 if d{xi,Xj) < d(xi,Xk), for 
all x k e x- 

A. Channel model and assumptions for the optimization prob- 
lem 

We consider a Complex Additive White Gaussian Noise 
(AWGN) channel with phase jitter, i.e., the received signal 
can be written as 

y — xe 3 ^ + n 

where n is a complex Gaussian random variable with concen- 
tration K n = l/er 2 , x is the transmitted constellation point 
and ip is the phase rotation. We assume that ip has a Tikhonov 
(also known as Von Mises) probability distribution function 
(PDF) with concentration K v , i.e., 



where 



P(<f) 



s (v) 



where Iq(.) is the modified Bessel function of the first kind of 
order zero. As we have mentioned we consider only the resid- 
ual phase error caused by the imperfect phase synchronization, 
and therefore the obtained results are independent from any 
specific phase-estimation technique. This simplification allows 
us to obtained a simple and fast approximation for the mutual 
information of any given constellation. 

Our goal is to find the constellation that maximizes AMI of 
the channel, when all the constellation points are transmitted 
with the same probability. Therefore we wish to solve the 
optimization problem 



X* = argmaxC(x) 

x 



(1) 



m — E, 
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where we used the short hand notation max* = log J exp. 
Note that in general one needs to solve the above optimization 
problem under some power constraints over the constellation 
space. In this paper we focus only on average power constraint 
and fix it to one. 

In order to calculate C(x) we need first to compute p(y\u) 



for any given input/ouput signals u and y 



-n. Since ip 



is assumed to be a random variable with Tikhonov distribution 
we have 



\og(p(y\u)) = log 



B 



p(y\u,ip')p((p')dip' 
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\{ip') + KJl(y*ue^') 



where we have introduced X((f) — K v cos(ip), and we have 
collected in the variable B all constant terms w.r.t. u. 

The main step to obtain a fast and accurate approximation 
of the mutual information is to substitute the max* operation 
in last row of (O with the maximum operation. Let z = y*u 
and f((p) — X((p) + Kn^ze^), to find the maximum we 
need to find the solutions of the equation g~ = resulting in 



dX(cp) 



dtp 



ip=<j> 



K n — SR(ze^; 
dtp 



v=<t> 



obtaining the following expression for 



sin(0) = -\z\— sin(argz + 0)), 

where is a function of z. Equation (0]i has the solution 

AS(z) 



(4) 



(z) = — arctan 



l + AR(af)' 



(5) 



where we have defined A = Substituting the above 



equation in ([3]) we obtain 



log(p(y\u)) * B - ~-f\u\ 2 



\((f>{y*u)) + Kn^ue^*^). 

The normalized logarithm in © can then be approximated as 

p(y\uY 



log 



Kn 

2 



p{y\x 

(2$t(y*ue 34,{y * u) ) - 2^{y*xe^ v ' x ^) 



\(<t>(y*u)) - \(<j>(y*x)) 



(6) 



Finally, in order to calculate the mutual information we need 
to first calculate the max* operation of © with respect to u 
and then take the average with respect to both n and ip in ©. 
To rapidly compute these averages we used the Gauss-Hermite 



quadrature formula with degree k = 7 per each of the three 
noise dimension fl6l . 

In the "pragmatic" receivers where the detector and the 
binary decoder are separated and no iterations take place 
between them the relevant measure to be optimized is the 
PAMI (see lfl5l ). For a given constellation \ an d labeling /i, 
the symmetric PAMI of the channel is defined as 

m 

Cp( X ,n) = J2 I & i ( X y< Y ) ^ C< 0C), (7) 
i=i 

where X is the transmitted signal, Y is the received symbol, 
and /(.; .) denotes the mutual information function. For PAMI, 
the loss in terms of channel capacity compared to optimal joint 
detection and decoding, not only depends on the constellation, 
but also on the labeling. In general, non-Gray mappings induce 
a higher loss of capacity at high SNR's. The approximated 
computation of (0 can be performed using the previously 
described approach with some slight modifications that we 
omit for brevity (for detail the reader is referred to Q). 

B. Simulated annealing algorithm for joint signal/labeling 
optimization 

In simulated annealing (SA) algorithm has been used 
for maximizing C(x) and Cp(x, fj) in the absence of the phase 
noise. We use the same algorithm for optimizing the constel- 
lation for phase noise channels. SA, under some conditions 
on the cooling schedule, guarantees convergence to the global 
optimum. However, these conditions are not usually feasible, 
as they impose a very slow cooling schedule. When these 
conditions are not satisfied, one may argue about the optimality 
of the obtained constellations. Therefore it is important to 
choose carefully the parameters of the SA algorithm in order 
to provide a good local optimum. 

The main adaptation needed to speed up the SA algorithm 
for maximizing the capacity is to define a maximum dis- 
placement length as a function of time. For details on such 
adaptation we refer the readers to [5 1 and the references within. 
The output of this algorithm -with the chosen parameters- is 
independent of the initial conditions for constellations with up 
to 16 signals, suggesting that the algorithm is optimal for such 
small constellations. However for larger constellations the SA 
algorithm becomes very slow and its output depends on the 
initial conditions. 

III. Optimization results 

In this section we present some of the 8 point optimized 
constellations and their corresponding capacity curves as a 
function of both SNR = K n /2 and the phase noise stan- 
dard deviation PNSD=<7 ¥ , = y/l/K^. We also compare 
the capacity curves of these optimized constellations with 
conventional 8-PSK constellation. As we will see, the shape 
of the optimized constellation strongly depends on both SNR 
and g v . Note that for each pair of a v and SNR values and 
the chosen objective function (AMI or PAMI) one needs to 
run the SA algorithm to find the optimized constellation. 
In figure Q] we plot both AMI and PAMI curves of the 



optimized constellations for several values of PNSD and values 
of SNR from 1 to 15. The solid and dashed curves correspond 
respectively to AMI and PAMI of the optimized constellations. 
As it can be noticed, the loss due to the pragmatic approach 
is quite small (less than 0.2 dB) in all cases. This is true also 
for higher order constellations. In Figure |2] we present the 
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Fig. 1 . AMI and PAMI for optimized constellation for various a v and SNR 
values. 

optimized 8-ary constellations for six different values of PNSD 
with the SNR fixed into 12 dB, and obtained maximizing 
the AMI. For comparison, in Figure [3] we also show two 
constellations obtained by optimizing PAMI. As it can be 
seen, the constellations are slightly different from those in the 
previous figure. 

To check the robustness of designed constellation to the mis- 
match with respect to the actual channel SNR in Figure [4] we 
report the AMI of the constellations of Figure [2] optimized for 
a target SNR of 12 dB, as a function of the actual channel SNR 
(solid lines). Comparing these curves with the ones in FigureQ] 
one can see that the constellations optimized at SNR=12 have 
performances close to the optimized constellations even fol- 
lower channel SNR values. However this is true only for small 
constellations. In Figure |4]we also plot the AMI curves for the 
constellation optimized at a v = (only AWGN) but used over 
the channel affected by phase noise (dashed lines). The loss 
due to the constellation suboptimality in this case is dramatic, 
showing that a constellation design that neglects the presence 
of phase noise can lead to very poor system performance. In 
particular, the loss increases with the target AMI to the point 
where it becomes infinite, as the AMI curves saturate below 
the maximum value of 3. 

To check the robustness of designed constellation to the 
mismatch with respect to the actual channel PNSD in Figure [5] 
we plot the AMI of the constellations of Figure [2] as a 
function of the channel PNSD. We also plot for reference 
the capacity curve for the 8-PSK constellation. Constellations 
designed for a target PNSD does not degrade their performance 
robustness when the channel PNSD is smaller, while their 
performance rapidly degrades when it becomes larger. Finally, 
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Fig. 2. Optimized 8-ary constellations for six different values of a v . The 
objective function is the AMI and the SNR is fixed into 12 dB in all cases. 
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Fig. 3. Optimized 8-ary constellations for two different values of a v . The 
objective function is the PAMI and the SNR is fixed to 12 dB in both cases. 
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Fig. 4. Performance of optimized constellations in Figure [2] as a function 
of SNR. The solid lines are the AMI for the optimized constellations for 
the given PNSD, the dashed-line curves are the AMI for the constellation 
optimized for the AWGN but used in the presence of the phase noise. 
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Fig. 5. Performance of optimized constellations in Figure [2] as a function of 
the phase noise standard deviation <j v . 



as it can be seen in Figure [5] the constellations optimized for 
high PNSD do not always saturate to the maximum spectral 
efficiency when used over channels with a lower phase noise 
standard deviation. Thus, there is a trade-off between the 
performance of the optimized constellations with respect to 
SNR and PNSD. Also we should notice that the optimized 
constellation over AWGN channel is not robust even for small 
channel PNSD values. This becomes particularly important 
when one needs to choose a constellation for a system where 
the PNSD is not accurately estimated, as it may cause a 
large loss in performance. These observations remain valid for 
larger constellations sets where the constellation performance 
becomes even more sensible to small changes of PNSD. 



IV. Simulation results 

In order to check the validity of our design approach, we 
have simulated the performance of the optimized constellations 
over a realistic system, using a powerful "capacity achieving" 
encoding scheme. In Figure [6] we report the results of the 
comparison of a set of 8 point constellations pragmatically 
encoded with a rate 5/6 binary serially concatenated convo- 
lutional code (SCCC). The details of the encoder structure 
can be found in ifTTl . We have considered the constellations 
reported in Figure [3] which have been optimized for the 
PAMI function fixing SNR=12 dB and PNSD=0 and 25 
degree. Looking at Figure |4] one can verify that this SNR 
corresponds to a PAMI of 2.5, thus justifying our choice 




Fig. 6. Performance comparison of several 8 points constellations using a 
rate 5/6 SCCC code. Dashed lines refer to the performance over the AWGN 
channel, solid lines refer to the AWGN channel affected by phase noise with 
PNSD=25 degrees. 



for the code rate (R — 5/6). Dashed curves refer to the 
AWGN channel without phase noise (channel PNSD=0). In 
this case the optimal constellation outperforms the 8-PSK 
constellation, reported for reference, by roughly 0.8 dB. The 
constellation designed for PNSD=25 shows a loss of about 0.3 
dB w.r.t. 8-PSK. Solid lines refer to the case of AWGN channel 
affected by phase noise with PNSD=25. The situation in this 
case drastically changes. 8-PSK performance is not capable 
of delivering reliable communication for any value of SNR 
and its performance curve is not reported. The constellation 
optimized for PNSD=0 shows a loss of 19 dB, while the 
optimal constellation for PNSD=25 shows a loss of only 2.8 
dB w.r.t. the AWGN channel and 3.9 dB w.r.t. the optimal 
constellation on the AWGN channel. Notice that these losses 
perfectly match those predicted by the PAMI computation in 
Figure |4] . Furthermore, to verify the difference between the 
PAMI and AMI optimization, in Figure [6] we also report the 
results obtained using a constellation optimized for the latter, 
associated to a Gray labeling. The constellation loses about 
0.2 dB. 

V. Conclusions 

In this paper we addressed the problem of constellation 
optimization for channels affected by phase noise. We first 
presented a fast approximated method to calculate the AMI 
of a given constellation for channels with white phase and 
thermal noise. Then the simulated annealing algorithm was 
used to maximize the AMI under the average power constraint. 
Our method allows also for a joint signal/labeling optimiza- 
tion of the constellation in pragmatic system scenarios, by 
maximizing the PAMI. We studied the performance of the 
optimized constellations both as a function of signal to noise 
ratio and of the phase noise standard deviation. We have also 
studied the robustness of our design to possible mismatches 
with respect to the actual channel conditions. In particular, it 
is observed that constellations optimized for AWGN channels 
alone are very sensitive to the phase noise, while other designs 



show more robustness to variations of PNSD and/or SNR. 
Finally to check the validity of our design approach we 
have simulated the optimized constellations over a realistic 
scenario including a capacity achieving code, obtaining results 
perfectly in line with the one predicted by AMI and PAMI 
calculations. This fact confirms that AMI and PAMI are the 
correct optimization functions to be considered for designing 
constellations to be embedded in system using turbo-codes. 
Even though we focused only on 8 point constellations, our 
method can be used to optimize much larger constellations. 
The results will be presented in an upcoming paper. 
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